[Fix](iceberg) Fix NoSuchElementException in IcebergSysTableJniScanner when reading empty system tables by zy-kkk · Pull Request #55379 · apache/doris

zy-kkk · 2025-08-27T11:49:09Z

Fix NoSuchElementException that occurs when querying Iceberg system tables with empty scan tasks by ensuring proper iterator state checking before calling reader.next().

Thearas · 2025-08-27T11:49:14Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zy-kkk · 2025-08-27T11:49:19Z

run buildall

suxiaogang223

LGTM

github-actions · 2025-08-27T11:54:02Z

PR approved by anyone and no changes requested.

github-actions · 2025-08-27T12:18:02Z

PR approved by at least one committer and no changes requested.

doris-robot · 2025-08-27T12:20:09Z

TPC-H: Total hot run time: 34711 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 237e013a228547704d44250774d014d5979719c2, data reload: false

------ Round 1 ----------------------------------
q1	17652	5380	5168	5168
q2	2004	323	245	245
q3	10219	1356	748	748
q4	10242	1045	536	536
q5	7547	2467	2400	2400
q6	193	169	138	138
q7	951	771	648	648
q8	9368	1461	1261	1261
q9	7010	5191	5212	5191
q10	6899	2399	1984	1984
q11	480	312	277	277
q12	364	369	236	236
q13	17772	3724	3011	3011
q14	238	239	224	224
q15	559	524	497	497
q16	447	435	380	380
q17	613	900	364	364
q18	7530	7166	7079	7079
q19	1097	967	611	611
q20	367	361	232	232
q21	4310	3278	2500	2500
q22	1099	1049	981	981
Total cold run time: 106961 ms
Total hot run time: 34711 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5267	5183	5236	5183
q2	260	333	234	234
q3	2258	2722	2352	2352
q4	1383	1846	1356	1356
q5	4265	4627	4605	4605
q6	237	178	137	137
q7	2172	1944	1809	1809
q8	2646	2729	2694	2694
q9	7443	7414	7322	7322
q10	3128	3389	2926	2926
q11	586	529	489	489
q12	711	805	608	608
q13	3651	3977	3453	3453
q14	271	338	312	312
q15	519	487	481	481
q16	482	506	472	472
q17	1194	1589	1410	1410
q18	7793	7813	7670	7670
q19	879	904	1005	904
q20	2011	2108	1941	1941
q21	5213	4546	4258	4258
q22	1101	1043	1004	1004
Total cold run time: 53470 ms
Total hot run time: 51620 ms

doris-robot · 2025-08-27T12:31:47Z

TPC-DS: Total hot run time: 187670 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 237e013a228547704d44250774d014d5979719c2, data reload: false

query1	1039	446	459	446
query2	6564	1812	1773	1773
query3	6764	236	225	225
query4	26850	23385	23236	23236
query5	4390	651	517	517
query6	355	244	223	223
query7	4650	544	299	299
query8	289	269	254	254
query9	8619	2938	2929	2929
query10	502	367	308	308
query11	15745	14988	14693	14693
query12	187	125	131	125
query13	1702	598	462	462
query14	9532	5932	5923	5923
query15	223	198	171	171
query16	7473	651	536	536
query17	1213	750	639	639
query18	2032	423	354	354
query19	237	203	187	187
query20	136	127	122	122
query21	227	132	112	112
query22	4030	4214	4015	4015
query23	33813	33020	32985	32985
query24	8194	2390	2424	2390
query25	589	523	453	453
query26	1254	290	173	173
query27	2725	528	360	360
query28	4412	2287	2272	2272
query29	806	611	502	502
query30	297	237	213	213
query31	915	803	750	750
query32	93	86	85	85
query33	591	395	365	365
query34	823	863	528	528
query35	870	856	739	739
query36	984	1026	904	904
query37	127	115	93	93
query38	4094	4076	4002	4002
query39	1484	1433	1446	1433
query40	226	142	141	141
query41	69	65	65	65
query42	125	116	119	116
query43	540	534	489	489
query44	1390	876	873	873
query45	183	174	181	174
query46	909	1027	670	670
query47	1775	1822	1729	1729
query48	410	438	333	333
query49	737	524	416	416
query50	691	717	422	422
query51	4193	4219	4107	4107
query52	118	118	113	113
query53	263	282	212	212
query54	629	627	570	570
query55	103	93	100	93
query56	366	360	344	344
query57	1201	1201	1132	1132
query58	327	289	282	282
query59	2722	2805	2588	2588
query60	374	357	359	357
query61	168	165	164	164
query62	823	769	672	672
query63	242	202	205	202
query64	4463	1163	884	884
query65	4320	4225	4221	4221
query66	1150	455	370	370
query67	15837	15383	15076	15076
query68	6978	958	676	676
query69	494	337	295	295
query70	1262	1186	1146	1146
query71	436	355	334	334
query72	5807	5102	5077	5077
query73	703	669	367	367
query74	9187	9109	8978	8978
query75	3160	3134	2651	2651
query76	3209	1154	801	801
query77	499	433	348	348
query78	9595	9816	8954	8954
query79	1636	818	599	599
query80	660	636	524	524
query81	511	259	236	236
query82	204	143	114	114
query83	265	271	258	258
query84	255	113	97	97
query85	855	468	438	438
query86	386	320	318	318
query87	4255	4287	4180	4180
query88	2898	2231	2219	2219
query89	400	336	290	290
query90	1936	234	226	226
query91	166	177	140	140
query92	100	77	74	74
query93	1620	1039	663	663
query94	708	428	350	350
query95	420	347	327	327
query96	497	601	279	279
query97	2611	2696	2555	2555
query98	257	228	225	225
query99	1326	1435	1320	1320
Total cold run time: 272480 ms
Total hot run time: 187670 ms

doris-robot · 2025-08-27T12:37:26Z

ClickBench: Total hot run time: 32.57 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 237e013a228547704d44250774d014d5979719c2, data reload: false

query1	0.05	0.05	0.05
query2	0.08	0.05	0.06
query3	0.24	0.08	0.08
query4	1.61	0.12	0.12
query5	0.45	0.42	0.43
query6	1.19	0.65	0.67
query7	0.04	0.03	0.03
query8	0.06	0.04	0.05
query9	0.60	0.54	0.51
query10	0.59	0.58	0.57
query11	0.17	0.12	0.12
query12	0.15	0.12	0.13
query13	0.62	0.64	0.61
query14	0.79	0.84	0.84
query15	0.87	0.86	0.86
query16	0.39	0.41	0.40
query17	1.04	1.07	1.11
query18	0.23	0.20	0.21
query19	1.95	1.85	1.79
query20	0.01	0.01	0.02
query21	15.42	0.98	0.58
query22	0.78	1.08	0.64
query23	15.08	1.42	0.60
query24	7.05	1.26	0.54
query25	0.46	0.18	0.17
query26	0.59	0.18	0.14
query27	0.07	0.05	0.06
query28	9.28	0.92	0.43
query29	12.54	3.91	3.22
query30	3.14	3.08	3.03
query31	2.82	0.62	0.40
query32	3.24	0.57	0.49
query33	3.09	3.17	3.11
query34	16.20	5.45	4.86
query35	4.97	4.89	4.93
query36	0.71	0.51	0.50
query37	0.10	0.07	0.08
query38	0.06	0.05	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.15
query41	0.08	0.03	0.03
query42	0.03	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 107.09 s
Total hot run time: 32.57 s

hello-stephen · 2025-08-27T12:45:38Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

…ge & fix systable read error (#55398) 1、Add iceberg rest test for different storage #54590 2、Fix NoSuchElementException that occurs when querying Iceberg system tables with empty scan tasks by ensuring proper iterator state checking before calling reader.next(). #55379

…r when reading empty system tables

zy-kkk · 2025-09-01T03:21:23Z

run buildall

doris-robot · 2025-09-01T03:56:35Z

TPC-H: Total hot run time: 34048 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b7aa78442f9ef9ddcaa5a5d0d70acf84789cbd85, data reload: false

------ Round 1 ----------------------------------
q1	17665	5226	5054	5054
q2	2008	315	215	215
q3	10263	1294	727	727
q4	10239	1020	542	542
q5	7517	2365	2356	2356
q6	179	174	138	138
q7	946	782	639	639
q8	9347	1336	1107	1107
q9	6904	5098	5179	5098
q10	6977	2392	1985	1985
q11	495	303	286	286
q12	341	357	234	234
q13	17781	3701	3065	3065
q14	241	259	216	216
q15	581	527	482	482
q16	441	437	383	383
q17	589	854	368	368
q18	7468	7146	7028	7028
q19	1375	952	566	566
q20	350	342	235	235
q21	3769	2554	2341	2341
q22	1078	1029	983	983
Total cold run time: 106554 ms
Total hot run time: 34048 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5129	5155	5118	5118
q2	253	327	227	227
q3	2200	2713	2375	2375
q4	1419	1819	1334	1334
q5	4224	4325	4674	4325
q6	215	175	133	133
q7	2022	2002	1903	1903
q8	2652	2658	2714	2658
q9	7336	7450	7514	7450
q10	3189	3443	2916	2916
q11	596	541	513	513
q12	765	836	644	644
q13	3740	3947	3463	3463
q14	286	294	298	294
q15	521	490	475	475
q16	466	502	430	430
q17	1151	1718	1435	1435
q18	7803	7673	7573	7573
q19	806	832	876	832
q20	2005	2197	2084	2084
q21	5033	4562	4468	4468
q22	1090	1072	1005	1005
Total cold run time: 52901 ms
Total hot run time: 51655 ms

doris-robot · 2025-09-01T04:08:09Z

TPC-DS: Total hot run time: 186421 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b7aa78442f9ef9ddcaa5a5d0d70acf84789cbd85, data reload: false

query1	1030	464	404	404
query2	6555	1737	1725	1725
query3	6794	231	232	231
query4	26221	23573	22879	22879
query5	4428	656	513	513
query6	324	251	251	251
query7	4649	531	305	305
query8	312	271	251	251
query9	8658	2947	2912	2912
query10	495	350	330	330
query11	15771	14950	15033	14950
query12	169	121	114	114
query13	1692	559	424	424
query14	9649	6005	5820	5820
query15	232	195	175	175
query16	7531	665	494	494
query17	1215	772	621	621
query18	2037	449	332	332
query19	205	192	180	180
query20	144	121	120	120
query21	219	140	176	140
query22	4171	4169	4050	4050
query23	33758	32846	33015	32846
query24	8115	2379	2431	2379
query25	563	532	452	452
query26	1250	277	167	167
query27	2732	513	358	358
query28	4380	2292	2262	2262
query29	800	589	491	491
query30	295	231	202	202
query31	935	838	710	710
query32	87	80	75	75
query33	584	408	355	355
query34	803	870	522	522
query35	807	833	763	763
query36	1004	993	917	917
query37	123	113	92	92
query38	4076	4111	4019	4019
query39	1502	1430	1433	1430
query40	224	136	130	130
query41	67	62	61	61
query42	130	119	129	119
query43	551	508	493	493
query44	1368	868	866	866
query45	182	177	172	172
query46	866	1017	663	663
query47	1777	1826	1737	1737
query48	390	450	329	329
query49	739	511	414	414
query50	663	689	406	406
query51	4089	4165	4062	4062
query52	124	116	110	110
query53	254	274	199	199
query54	627	607	543	543
query55	100	93	95	93
query56	353	348	341	341
query57	1195	1228	1154	1154
query58	309	290	290	290
query59	2761	2724	2533	2533
query60	370	352	343	343
query61	162	159	159	159
query62	818	733	671	671
query63	232	196	196	196
query64	4531	1146	866	866
query65	4313	4219	4221	4219
query66	1146	444	337	337
query67	15860	15262	15258	15258
query68	7953	935	598	598
query69	514	342	302	302
query70	1196	1074	1160	1074
query71	420	338	324	324
query72	5435	4980	5020	4980
query73	664	635	359	359
query74	8957	9087	8734	8734
query75	3153	3074	2618	2618
query76	3235	1126	737	737
query77	521	407	343	343
query78	9579	9931	8874	8874
query79	1365	836	611	611
query80	694	587	588	587
query81	496	266	222	222
query82	204	138	118	118
query83	266	264	257	257
query84	256	111	91	91
query85	848	456	431	431
query86	336	322	283	283
query87	4290	4286	4210	4210
query88	2841	2220	2216	2216
query89	402	336	300	300
query90	1759	219	221	219
query91	156	170	134	134
query92	80	73	69	69
query93	1085	1023	654	654
query94	652	474	329	329
query95	417	334	327	327
query96	489	593	271	271
query97	2607	2725	2586	2586
query98	244	220	220	220
query99	1352	1465	1318	1318
Total cold run time: 270982 ms
Total hot run time: 186421 ms

doris-robot · 2025-09-01T04:13:47Z

ClickBench: Total hot run time: 32.51 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b7aa78442f9ef9ddcaa5a5d0d70acf84789cbd85, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.06
query3	0.25	0.08	0.09
query4	1.60	0.12	0.12
query5	0.46	0.42	0.43
query6	1.19	0.64	0.66
query7	0.04	0.03	0.03
query8	0.05	0.04	0.04
query9	0.61	0.52	0.52
query10	0.58	0.59	0.58
query11	0.16	0.12	0.11
query12	0.16	0.13	0.12
query13	0.64	0.62	0.62
query14	0.80	0.84	0.84
query15	0.87	0.84	0.86
query16	0.39	0.40	0.38
query17	1.06	1.05	1.02
query18	0.22	0.20	0.20
query19	1.91	1.82	1.83
query20	0.02	0.01	0.01
query21	15.39	0.94	0.58
query22	0.76	1.13	0.72
query23	14.95	1.40	0.65
query24	7.23	0.73	0.92
query25	0.47	0.11	0.08
query26	0.59	0.16	0.13
query27	0.06	0.06	0.06
query28	9.73	0.90	0.42
query29	12.62	3.95	3.20
query30	3.06	3.00	2.94
query31	2.81	0.57	0.38
query32	3.25	0.54	0.47
query33	3.09	3.20	3.07
query34	16.14	5.46	4.82
query35	4.88	4.94	4.91
query36	0.71	0.51	0.49
query37	0.10	0.08	0.07
query38	0.05	0.05	0.04
query39	0.04	0.03	0.03
query40	0.18	0.14	0.14
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 107.43 s
Total hot run time: 32.51 s

zy-kkk · 2025-09-01T06:40:01Z

Tested in #55484 and #54590

suxiaogang223 approved these changes Aug 27, 2025

View reviewed changes

github-actions bot added the reviewed label Aug 27, 2025

CalvinKirs approved these changes Aug 27, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 27, 2025

morningman added the dev/3.1.x label Aug 27, 2025

morningman approved these changes Aug 27, 2025

View reviewed changes

morningman approved these changes Aug 28, 2025

View reviewed changes

zy-kkk mentioned this pull request Aug 28, 2025

branch-3.1: [test](iceberg) Add iceberg rest test for different storage & fix systable read error #55398

Merged

morningman added dev/3.1.0-merged and removed dev/3.1.x labels Aug 28, 2025

[Fix](iceberg) Fix NoSuchElementException in IcebergSysTableJniScanne…

b7aa784

…r when reading empty system tables

zy-kkk force-pushed the fxi_iceberg_sys_table_scan branch from 237e013 to b7aa784 Compare September 1, 2025 03:21

morningman approved these changes Sep 1, 2025

View reviewed changes

zy-kkk merged commit 0cb019e into apache:master Sep 1, 2025
25 of 26 checks passed

zy-kkk deleted the fxi_iceberg_sys_table_scan branch September 1, 2025 06:40

Conversation

zy-kkk commented Aug 27, 2025

Uh oh!

Thearas commented Aug 27, 2025

Uh oh!

zy-kkk commented Aug 27, 2025

Uh oh!

suxiaogang223 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 27, 2025

Uh oh!

github-actions bot commented Aug 27, 2025

Uh oh!

doris-robot commented Aug 27, 2025

Uh oh!

doris-robot commented Aug 27, 2025

Uh oh!

doris-robot commented Aug 27, 2025

Uh oh!

hello-stephen commented Aug 27, 2025

FE UT Coverage Report

Uh oh!

zy-kkk commented Sep 1, 2025

Uh oh!

doris-robot commented Sep 1, 2025

Uh oh!

doris-robot commented Sep 1, 2025

Uh oh!

doris-robot commented Sep 1, 2025

Uh oh!

zy-kkk commented Sep 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants